Perception of Synthesized Audible and Visible Speech

The research reported in this paper uses novel stimuli to study how speech perception is influenced by information presented to ear and eye. Auditory and visual sources ofinformation (syllables) were synthesized and presented in isolation or in factorial combination. A five-step contilllium between the syllables /bal and Idalwas synthesized along both auditory and visual dimensions, by varying properties of the syllable at its onset. The onsets of the second and third formants were manipulatedin the audible speech. For the visible speech, the shape oft/Ie lips and the jaw position at the onset ofthe syllable were manipulated. Subjects' identification jildgments of the test syllables presented on videotape were influenced by both auditory and visual information. The results.were used to test between a fuzzy logical model of speech perception (FLMP) and a categorical model of perception (CMP). These tests indicate that evaluation and integration of the two sources of information makes available continuous as opposed tojust categorical information. In addition, the integration of the two sources appears to be nonadditive in that the least ambiguous source has the largest impact on thejudgment. The two sources of information appear to be evaluated, integrated. and identified as described by the FLMP-an optimal algorithm for combining information from multiple sources. The research provides a theoretical framework for understanding the improvement in speech perception by hearing-impaired listeners when auditory speech is supplemented with other sources of information.